Optimization Tutorial 2 : Newton ’ s Method , Karush - Kuhn - Tucker ( KKT ) Conditions 3 3 Constrained Optimization and KKT Optimality Conditions

In the first part of the tutorial, we introduced the problem of unconstrained optimization, provided necessary and sufficient conditions for optimality of a solution to this problem, and described the gradient descent method for finding a (locally) optimal solution to a given unconstrained optimization problem. We now describe another method for unconstrained optimization, namely Newton’s method, that has better convergence guarantees than the gradient descent method in many settings. We will then start with the more challenging problem of constrained optimization, where we shall look at the Karush-Kuhn-Tucker (KKT) conditions for optimality of a solution to a constrained optimization problem.